Spontaneous perspective-taking in real-time language comprehension: evidence from eye-movements and grain of coordination.

Linguistic communication requires interlocutors to consider differences in each other's knowledge (perspective-taking). However, perspective-taking might either be spontaneous or strategic. We monitored listeners' eye movements in a referential communication task. A virtual speaker gave temporally ambiguous instructions with scalar adjectives ("big" in "big cubic block"). Scalar adjectives assume a contrasting object (a small cubic block). We manipulated whether the contrasting object (a small triangle) for a competitor object (a big triangle) was in common ground (visible to both speaker and listener) or was occluded so it was in the listener's privileged ground, in which case perspective-taking would allow earlier reference-resolution. We used a complex visual context with multiple objects, making strategic perspective-taking unlikely when all objects are in the listener's referential domain. A turn-taking, puzzle-solving task manipulated whether participants could anticipate a more restricted referential domain. Pieces were either confined to a small area (requiring fine-grained coordination) or distributed across spatially distinct regions (requiring only coarse-grained coordination). Results strongly supported spontaneous perspective-taking: Although comprehension was less time-locked in the coarse-grained condition, participants in both conditions used perspective information to identify the target referent earlier when the competitor contrast was in privileged ground, even when participants believed instructions were computer-generated.

Musical coordination affects children's perspective-taking, but musical synchrony does not.

Perspective-taking, which is important for communication and social activities, can be cultivated through joint actions, including musical activities in children. We examined how rhythmic activities requiring coordination affect perspective-taking in a referential communication task with 100 Chinese 4- to 6-year-old children. In Study 1, 5- to 6-year-old children played an instrument with a virtual partner in one of three coordination conditions: synchrony, asynchrony, and antiphase synchrony. Eye movements were then monitored with the partner giving instructions to identify a shape referent which included a pre-nominal scalar adjective (e.g., big cubic block). When the target contrast (a small cubic block) was in the shared ground and a competitor contrast was occluded for the partner, participants who used perspective differences could, in principle, identify the intended referent before the shape was named. We hypothesized that asynchronous and antiphase synchronous musical activities, which require self-other distinction, might have stronger effects on perspective-taking than synchronous activity. Children in the asynchrony and antiphase synchrony conditions, but not the synchrony condition, showed anticipatory looks at the target, demonstrating real-time use of the partner's perspective. Study 2 was conducted to determine if asynchrony and antiphase asynchrony resulted in perspective-taking that otherwise would not have been observed, or if synchronous coordination inhibited perspective-taking that would otherwise have occurred. We found no evidence for online perspective-taking in 4- to 6-year-old children without music manipulation. Therefore, playing instruments asynchronously or in alternation, but not synchronously, increases perspective-taking in children of this age, likely by training self-other distinction and control. A video abstract of this article can be viewed at https://youtu.be/TM9h_GpFlsA. RESEARCH HIGHLIGHTS: This study is the first to show that rhythmic coordination, a form of non-linguistic interaction, can affect children's performance in a subsequent linguistic task. Eye-movement data revealed that children's perspective-taking in language processing was facilitated by prior asynchronous and antiphase synchronous musical interactions, but not by synchronous coordination. The results challenge the common "similar is better" view, suggesting that maintaining self-other distinction may benefit social interactions that involve representing individual differences.

Syllable Inference as a Mechanism for Spoken Language Understanding.

A classic problem in spoken language comprehension is how listeners perceive speech as being composed of discrete words, given the variable time-course of information in continuous signals. We propose a syllable inference account of spoken word recognition and segmentation, according to which alternative hierarchical models of syllables, words, and phonemes are dynamically posited, which are expected to maximally predict incoming sensory input. Generative models are combined with current estimates of context speech rate drawn from neural oscillatory dynamics, which are sensitive to amplitude rises. Over time, models which result in local minima in error between predicted and recently experienced signals give rise to perceptions of hearing words. Three experiments using the visual world eye-tracking paradigm with a picture-selection task tested hypotheses motivated by this framework. Materials were sentences that were acoustically ambiguous in numbers of syllables, words, and phonemes they contained (cf. English plural constructions, such as "saw (a) raccoon(s) swimming," which have two loci of grammatical information). Time-compressing, or expanding, speech materials permitted determination of how temporal information at, or in the context of, each locus affected looks to, and selection of, pictures with a singular or plural referent (e.g., one or more than one raccoon). Supporting our account, listeners probabilistically interpreted identical chunks of speech as consistent with a singular or plural referent to a degree that was based on the chunk's gradient rate in relation to its context. We interpret these results as evidence that arriving temporal information, judged in relation to language model predictions generated from context speech rate evaluated on a continuous scale, informs inferences about syllables, thereby giving rise to perceptual experiences of understanding spoken language as words separated in time.

Predictions of Miscommunication in Verbal Communication During Collaborative Joint Action.

Purpose The purpose of the current study was to examine the lexical and pragmatic factors that may contribute to turn-by-turn failures in communication (i.e., miscommunication) that arise regularly in interactive communication. Method Using a corpus from a collaborative dyadic building task, we investigated what differentiated successful from unsuccessful communication and potential factors associated with the choice to provide greater lexical information to a conversation partner. Results We found that more successful dyads' language tended to be associated with greater lexical density, lower ambiguity, and fewer questions. We also found participants were more lexically dense when accepting and integrating a partner's information (i.e., grounding) but were less lexically dense when responding to a question. Finally, an exploratory analysis suggested that dyads tended to spend more lexical effort when responding to an inquiry and used assent language accurately-that is, only when communication was successful. Conclusion Together, the results suggest that miscommunication both emerges and benefits from ambiguous and lexically dense utterances.

Effects of coordination and gender on prosocial behavior in 4-year-old Chinese children.

In a block-assembly task with 138, 4-year-old Chinese kindergarten children, tested in pairs, we manipulated whether fine-grained coordination was required for accomplishing a shared goal with the same end product: building two adjoined towers with alternating levels of orange and green colored blocks to match a depicted model. In the coordination condition, each child had blocks of only one color and built the towers together. In the shared-goal-only condition, each child had both color blocks and built one of the towers, which they then adjoined. We predicted that children in the coordination condition would be more prosocial than children in the shared-goal-only condition. Studies with Western children typically find that girls are more generous than boys. However, we predicted the opposite pattern because Chinese culture emphasizes the importance of generosity more for males than females. Children in the coordination condition were more willing to help their partner complete an unrelated task and were more generous in sharing stickers with unknown children in a dictator game. These results demonstrate that level of coordination affects prosociality above and beyond having a shared goal, and are the first demonstration that prosocial effects of a collaborative task with children generalize beyond the participants to anonymous strangers. Boys shared more stickers with unknown children than girls, suggesting that gender differences in generosity are, in part, culturally conditioned.

Effects of distributional information on categorization of prosodic contours.

Although prosody clearly affects the interpretation of utterances, the mapping between prosodic representations and acoustic features is highly variable. Listeners may in part cope with this variability by adapting to distributions of acoustic features in the input. We examined whether listeners adapt to distributional changes using the construction It looks like an X. When pronounced with an H* pitch accent on the final noun and a low boundary tone, the construction supports an affirmative interpretation (e.g., It looks like a ZEBRA [and I think it is one]). Conversely, when pronounced with a L+H* pitch accent and a rising boundary tone, it suggests a negative interpretation (e.g., It LOOKS like a zebra.... [but it is not]). Experiment 1 elicited pragmatic interpretations of resynthesized 12-step continua with these two contours as the end points. In Experiment 2, one group of listeners heard items sampled from the most ambiguous region along the continua followed by affirmative continuations (e.g., It looks like a zebra because it has stripes all over its body) and items near the contrastive endpoint followed by negative continuations (e.g., It looks like a zebra but it is actually something else). Another group heard the reverse (i.e., ambiguous items with negative continuations and non-contrastive items with affirmative continuations). The two groups of participants subsequently derived diverging interpretations for novel ambiguous items, suggesting that prosodic processing involves flexible mappings between acoustic features and prosodic representations that are meaningful in interpretation of speech.

The effect of character contextual diversity on eye movements in Chinese sentence reading.

Chen, Huang, et al. (Psychonomic Bulletin & Review, 2017) found that when reading two-character Chinese words embedded in sentence contexts, contextual diversity (CD), a measure of the proportion of texts in which a word appears, affected fixation times to words. When CD is controlled, however, frequency did not affect reading times. Two experiments used the same experimental designs to examine whether there are frequency effects of the first character of two-character words when CD is controlled. In Experiment 1, yoked triples of characters from a control group, a group matched for character CD that is lower in frequency, and a group matched in frequency with the control group, but higher in character CD, were rotated through the same sentence frame. In Experiment 2 each character from a larger set was embedded in a separate sentence frame, allowing for a larger difference in log frequency compared to Experiment 1 (0.8 and 0.4, respectively). In both experiments, early and later eye movement measures were significantly shorter for characters with higher CD than for characters with lower CD, with no effects of character frequency. These results place constraints on models of visual word recognition and suggest ways in which Chinese can be used to tease apart the nature of context effects in word recognition and language processing in general.

The effect of contextual diversity on eye movements in Chinese sentence reading.

Recent studies have demonstrated that when contextual diversity is controlled token word frequency has minimal effects on visual word recognition. With the exception of a single experiment by Plummer, Perea, & Rayner (2014, Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 275-283), those studies have examined words in isolation. The current studies address two potential limitations of the Plummer et al. experiment. First, because Plummer et al. used different sentence frames for words in different conditions, the effects might be due to uncontrolled differences on the sentences. Second, the absence of a frequency effect might be attributed to comparing higher and lower frequency words within a limited range. Three eye-tracking experiments examined effects of contextual diversity and frequency on Mandarin Chinese, a logographic language, for words embedded in the normal sentences. In Experiment 1, yoked words were rotated through the same sentence frame. Experiments 2a and 2b used a design similar to Plummer et al., which allows use of a larger sample of words to compare results between experiments with a smaller and larger difference in log frequency (0.41 and 1.06, respectively). In all three experiments, first-pass and later eye movement measures were significantly shorter for targets with higher contextual diversity than for targets with lower contextual diversity, with no effects of frequency.

Now or later: Perceptual data are not immediately forgotten during language processing.

Christiansen & Chater (C&C) propose that language comprehenders must immediately compress perceptual data by "chunking" them into higher-level categories. Effective language understanding, however, requires maintaining perceptual information long enough to integrate it with downstream cues. Indeed, recent results suggest comprehenders do this. Although cognitive systems are undoubtedly limited, frameworks that do not take into account the tasks that these systems evolved to solve risk missing important insights.

Dynamically adapted context-specific hyper-articulation: Feedback from interlocutors affects speakers' subsequent pronunciations.

We ask whether speakers can adapt their productions when feedback from their interlocutors suggests that previous productions were perceptually confusable. To address this question, we use a novel web-based task-oriented paradigm for speech recording, in which participants produce instructions towards a (simulated) partner with naturalistic response times. We manipulate (1) whether a target word with a voiceless plosive (e.g., pill) occurs in the presence of a voiced competitor (bill) or an unrelated word (food) and (2) whether or not the simulated partner occasionally misunderstands the target word. Speakers hyper-articulated the target word when a voiced competitor was present. Moreover, the size of the hyper-articulation effect was nearly doubled when partners occasionally misunderstood the instruction. A novel type of distributional analysis further suggests that hyper-articulation did not change the target of production, but rather reduced the probability of perceptually ambiguous or confusable productions. These results were obtained in the absence of explicit clarification requests, and persisted across words and over trials. Our findings suggest that speakers adapt their pronunciations based on the perceived communicative success of their previous productions in the current environment. We discuss why speakers make adaptive changes to their speech and what mechanisms might underlie speakers' ability to do so.

Prosodic expectations in silent reading: ERP evidence from rhyme scheme and semantic congruence in classic Chinese poems.

In an ERP study, classic Chinese poems with a well-known rhyme scheme were used to generate an expectation of a rhyme in the absence of an expectation for a specific character. Critical characters were either consistent or inconsistent with the expected rhyme scheme and semantically congruent or incongruent with the content of the poem. These stimuli allowed us to examine whether a top-down rhyme scheme expectation would affect relatively early components of the ERP associated with character-to-sound mapping (P200) and lexically-mediated semantic processing (N400). The ERP data revealed that rhyme scheme congruence, but not semantic congruence modulated the P200: rhyme-incongruent characters elicited a P200 effect across the head demonstrating that top-down expectations influence early phonological coding of the character before lexical-semantic processing. Rhyme scheme incongruence also produced a right-lateralized N400-like effect. Moreover, compared to semantically congruous poems, semantically incongruous poems produced a larger N400 response only when the character was consistent with the expected rhyme scheme. The results suggest that top-down prosodic expectations can modulate early phonological processing in visual word recognition, indicating that prosodic expectations might play an important role in silent reading. They also suggest that semantic processing is influenced by general knowledge of text genre.

The Flexibility of Conceptual Pacts: Referring Expressions Dynamically Shift to Accommodate New Conceptualizations.

In a classic paper, Brennan and Clark argued that when interlocutors agree on a name for an object, they are forming a temporary agreement on how to conceptualize that object; that is, they are forming a conceptual pact. The literature on conceptual pacts has largely focused on the costs and benefits of breaking and maintaining lexical precedents, and the degree to which they might be partner-specific. The research presented here focuses on a question about conceptual pacts that has been largely neglected in the literature: To what extent are conceptual pacts specific to the local context of the interaction? If conceptual pacts are indeed temporary, then when the local context changes in ways that are accessible to participants, we would expect participants to seamlessly shift to referential expressions that reflect novel conceptualizations. Two experiments examined how referential forms change across context in collaborative, task-oriented dialog between naïve participants. In Experiment 1, names for parts of an unknown object were established in an "item" identification stage (e.g., a shape that looked like a wrench was called "the wrench"). In a second "build" stage, that name was often supplanted by an object-oriented name, e.g., the "leg." These changes happened abruptly and without negotiation. In Experiment 2, interlocutors manipulated clip art and more abstract tangram pictures in a "slider" puzzle to arrange the objects into a target configuration. On some trials moving an object revealed a picture that could be construed as a contrast competitor, e.g., a clip art picture of a camel after "the camel" had been negotiated as a name for a tangram shape, or vice versa. As would be expected, modification rates increased when a potential contrast was revealed. More strikingly, the degree to which a name had been negotiated or the frequency with which it had been used did not affect the likelihood that the revealed shape would be considered as a potential contrast. We find little evidence that names that are introduced as part of a conceptual pact persist when either the task goals or informational needs change. Rather, conceptual pacts are fluid temporary agreements.

What's in a Name? Interlocutors Dynamically Update Expectations about Shared Names.

In order to refer using a name, speakers must believe that their addressee knows about the link between the name and the intended referent. In cases where speakers and addressees learned a subset of names together, speakers are adept at using only the names their partner knows. But speakers do not always share such learning experience with their conversational partners. In these situations, what information guides speakers' choice of referring expression? A speaker who is uncertain about a names' common ground (CG) status often uses a name and description together. This N+D form allows speakers to demonstrate knowledge of a name, and could provide, even in the absence of miscommunication, useful evidence to the addressee regarding the speaker's knowledge. In cases where knowledge of one name is associated with knowledge of other names, this could provide indirect evidence regarding knowledge of other names that could support generalizations used to update beliefs about CG. Using Bayesian approaches to language processing as a guiding framework, we predict that interlocutors can use their partner's choice of referring expression, in particular their use of an N+D form, to generate more accurate beliefs regarding their partner's knowledge of other names. In Experiment 1, we find that domain experts are able to use their partner's referring expression choices to generate more accurate estimates of CG. In Experiment 2, we find that interlocutors are able to infer from a partner's use of an N+D form which other names that partner is likely to know or not know. Our results suggest that interlocutors can use the information conveyed in their partner's choice of referring expression to make generalizations that contribute to more accurate beliefs about what is shared with their partner, and further, that models of CG for reference need to account not just for the status of referents, but the status of means of referring to those referents.

Talker-specificity and adaptation in quantifier interpretation.

Linguistic meaning has long been recognized to be highly context-dependent. Quantifiers like many and some provide a particularly clear example of context-dependence. For example, the interpretation of quantifiers requires listeners to determine the relevant domain and scale. We focus on another type of context-dependence that quantifiers share with other lexical items: talker variability. Different talkers might use quantifiers with different interpretations in mind. We used a web-based crowdsourcing paradigm to study participants' expectations about the use of many and some based on recent exposure. We first established that the mapping of some and many onto quantities (candies in a bowl) is variable both within and between participants. We then examined whether and how listeners' expectations about quantifier use adapts with exposure to talkers who use quantifiers in different ways. The results demonstrate that listeners can adapt to talker-specific biases in both how often and with what intended meaning many and some are used.

Availability of Alternatives and the Processing of Scalar Implicatures: A Visual World Eye-Tracking Study.

Two visual world experiments investigated the processing of the implicature associated with some using a "gumball paradigm." On each trial, participants saw an image of a gumball machine with an upper chamber with orange and blue gumballs and an empty lower chamber. Gumballs dropped to the lower chamber, creating a contrast between a partitioned set of gumballs of one color and an unpartitioned set of the other. Participants then evaluated spoken statements, such as "You got some of the blue gumballs." Experiment 1 investigated the time course of the pragmatic enrichment from some to not all when the only utterance alternatives available to refer to the different sets were some and all. In Experiment 2, the number terms two, three, four, and five were also included in the set of alternatives. Scalar implicatures were delayed relative to the interpretation of literal statements with all only when number terms were available. The results are interpreted as evidence for a constraint-based account of scalar implicature processing.

Inferring Difficulty: Flexibility in the Real-time Processing of Disfluency.

Upon hearing a disfluent referring expression, listeners expect the speaker to refer to an object that is previously unmentioned, an object that does not have a straightforward label, or an object that requires a longer description. Two visual-world eye-tracking experiments examined whether listeners directly associate disfluency with these properties of objects, or whether disfluency attribution is more flexible and involves situation-specific inferences. Since in natural situations reference to objects that do not have a straightforward label or that require a longer description is correlated with both production difficulty and with disfluency, we used a mini-artificial lexicon to dissociate difficulty from these properties, building on the fact that recently learned names take longer to produce than existing words in one's mental lexicon. The results demonstrate that disfluency attribution involves situation-specific inferences; we propose that in new situations listeners spontaneously infer what may cause production difficulty. However, the results show that these situation-specific inferences are limited in scope: listeners assessed difficulty relative to their own experience with the artificial names, and did not adapt to the assumed knowledge of the speaker.

Form-to-expectation matching effects on first-pass eye movement measures during reading.

Recent Electroencephalography/Magnetoencephalography (EEG/MEG) studies suggest that when contextual information is highly predictive of some property of a linguistic signal, expectations generated from context can be translated into surprisingly low-level estimates of the physical form-based properties likely to occur in subsequent portions of the unfolding signal. Whether form-based expectations are generated and assessed during natural reading, however, remains unclear. We monitored eye movements while participants read phonologically typical and atypical nouns in noun-predictive contexts (Experiment 1), demonstrating that when a noun is strongly expected, fixation durations on first-pass eye movement measures, including first fixation duration, gaze duration, and go-past times, are shorter for nouns with category typical form-based features. In Experiments 2 and 3, typical and atypical nouns were placed in sentential contexts normed to create expectations of variable strength for a noun. Context and typicality interacted significantly at gaze duration. These results suggest that during reading, form-based expectations that are translated from higher-level category-based expectancies can facilitate the processing of a word in context, and that their effect on lexical processing is graded based on the strength of category expectancy.

Context-driven expectations about focus alternatives.

What is conveyed by a sentence frequently depends not only on the descriptive content carried by its words, but also on implicit alternatives determined by the context of use. Four visual world eye-tracking experiments examined how alternatives are generated based on aspects of the discourse context and used in interpreting sentences containing the focus operators only and also. Experiment 1 builds on previous reading time studies showing that the interpretations of only sentences are constrained by alternatives explicitly mentioned in the preceding discourse, providing fine-grained time course information about the expectations triggered by only. Experiments 2 and 3 show that, in the absence of explicitly mentioned alternatives, lexical and situation-based categories evoked by the context are possible sources of alternatives. While Experiments 1-3 all demonstrate the discourse dependence of alternatives, only explicit mention triggered expectations about alternatives that were specific to sentences with only. By comparing only with also, Experiment 4 begins to disentangle expectations linked to the meanings of specific operators from those generalizable to the class of focus-sensitive operators. Together, these findings show that the interpretation of sentences with focus operators draws on both dedicated mechanisms for introducing alternatives into the discourse context and general mechanisms associated with discourse processing.

Anticipatory Deaccenting in Language Comprehension.

We evaluated the hypothesis that listeners can generate expectations about upcoming input using anticipatory deaccenting, in which the absence of a nuclear pitch accent on an utterance-new noun is licensed by the subsequent repetition of that noun (e.g. Drag the SQUARE with the house to the TRIangle with the house). The phonemic restoration paradigm was modified to obscure word-initial segmental information uniquely identifying the final word in a spoken instruction, resulting in a stimulus compatible with two lexical alternatives (e.g. mouse/house). In Experiment 1, we measured participants' final interpretations and response times. Experiment 2 used the same materials in a crowd-sourced gating study. Sentence interpretations at gated intervals, final interpretations, and response times provided converging evidence that the anticipatory deaccenting pattern contributed to listeners' referential expectations. The results illustrate the availability and importance of sentence-level accent patterns in spoken language comprehension.